feat: add runtime system and gesture commands#415
Conversation
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ee2e42137c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const target = options.target ?? { kind: 'viewport' as const }; | ||
| if (target.kind === 'viewport') return { kind: 'viewport' }; |
There was a problem hiding this comment.
resolveScrollTarget returns a viewport target immediately when target is omitted or set to viewport, so this path never runs through resolveInteractionTarget (which is where assertSupportedInteractionSurface is enforced). On macOS desktop/menubar sessions, interactions.scroll can therefore execute backend gestures even though those surfaces are intended to be inspect-only, leading to unintended host-level scrolling.
Useful? React with 👍 / 👎.
| const capture = await captureInteractionSnapshot(runtime, options, false); | ||
| const viewport = resolveSnapshotViewport(capture.snapshot.nodes); |
There was a problem hiding this comment.
When swipe is invoked with a direction but no explicit from, resolveSwipeFrom goes straight to snapshot capture and viewport-center calculation without checking the interaction surface. That bypasses the macOS desktop/menubar guard used by target-resolved interactions, so unsupported swipes can be sent to inspect-only surfaces.
Useful? React with 👍 / 👎.
| const centerTarget = options.center | ||
| ? await resolveInteractionTarget( |
There was a problem hiding this comment.
pinchCommand only calls resolveInteractionTarget when options.center is provided, but immediately invokes backend.pinch when center is omitted. Since surface restrictions are enforced in target resolution, interactions.pinch({ scale }) bypasses the macOS desktop/menubar safety gate and can run gestures on sessions that should remain inspect-only.
Useful? React with 👍 / 👎.
| const rects = nodes.map((node) => node.rect).filter(isUsableRect); | ||
| if (rects.length === 0) { | ||
| throw new AppError('COMMAND_FAILED', 'Cannot infer viewport for directional swipe'); | ||
| } | ||
| const minX = Math.min(...rects.map((rect) => rect.x)); |
There was a problem hiding this comment.
resolveSnapshotViewport computes bounds from every node rect in the snapshot. Because snapshots can contain off-screen elements (for example, deep list content), these aggregate min/max bounds may place the derived center outside the visible frame, so direction-only swipes start off-screen and often no-op or hit the wrong area.
Useful? React with 👍 / 👎.
thymikee
force-pushed
the
codex/runtime-system-gestures
branch
from
ee2e421 to
457147a
Compare
thymikee
force-pushed
the
codex/runtime-system-gestures
branch
from
457147a to
32c247d
Compare
1 participant
Summary
Adds runtime-owned phase 2 system/device commands and phase 3 gesture commands behind typed backend primitives.
Splits interaction target resolution into a shared command module so click/fill and the new focus/longPress/swipe/scroll/pinch commands use the same selector/ref/point semantics. Adds system result unions, router bindings, catalog entries, conformance coverage, runtime tests, and docs/skill notes for clipboard sensitivity and macOS inspect-only gesture limits.
Touched-file count: 17. Scope stayed within the planned runtime command/backend/conformance modules plus targeted docs/skill notes requested during review.
Validation
pnpm formatpnpm test:unit src/__tests__/runtime-interactions.test.tspnpm check:quickpnpm check:unit